Flask and VueJs are widely known by many developers around the world. Flask is a popular Python Frameworks, and Vue is Javascript Popular Frameworks. As I mention about backend and frontend before, basically I build an application which consists of two sub-application, backend, and frontend. I use Flask as a tool to build the backend application and Vue to build the frontend application.

The main purpose of this application is to crawl tweets by a hashtag, determine the sentiment, and show it on a dashboard.

Following is the step that I do when building this application.

I use Python 3.6 and Vue 2.0 to build our application.

Now, let’s start to code.

I use Jupyter Notebook as a tool to develop the Model, it helps me a lot when preprocessing the train data and to build the classification model. I use Naive Bayes because this is the simpler approach to classify the sentiment of a tweet.

You can download the train.csv as training data from https://www.kaggle.com/c/twitter-sentiment-analysis2/data

First, I make a notebook named Preparing.ipynb to prepare the tweets data and then import the Python libraries that we need and then read the CSV train data.

I use pandas to handle dataframe, re also nltk to help me preprocess and clean the tweets data. To check the data, whether it is loaded or not, I usually check using tweet_df.head().

I only use the Sentiment and SentimentText column to train the data. I ignore itemID because I think this is no use for me. Based on the download source of our training data, Sentiment value 0 means Negative Sentiment, and 1 means Positive Sentiment.

I use the negation list to add NOT_ to every text after a negation until the following punctuation based on what I read in this document (https://web.stanford.edu/class/cs124/lec/sentiment.pdf)

line 7 until 11 on preProcessTweet function are used to lowercased tweet, remove punctuation from negation, remove web URL, remove user mention, and then remove hashtag. The next step is to stem every word in a tweet, add NOT_ as explained above, and then removing stop words.

Use this syntax to apply the preProcessTweets function in our dataframe. Use the head() function to check the new column(clean_text) that contains preprocessed tweets.

The above syntax will run for a while based on specifications of our hardware, save it in a new CSV file, so we don't need to run the preprocess function again.

Now let’s train the Model using the Naive Bayes Classification algorithm. I use Scikit-Learn to train the data.

I used a different notebook to train the Model. I named it modeling.ipynb. Of course, the first step in this new notebook is to import all of the libraries that we need later.

Load the preprocessed tweets.

I don't know if you face the data types error or not, but if you face it, you can use this syntax to convert the object into String/Unicode.

Then I made a pipeline that automates the training process. This pipeline is used to create a bag of words model and then train the classifier.

Split the data into train and test set, I only split it using 70:30 proportions and then train the data using the pipeline that I made before. Print the classification report to know the accuracy, recall, and precision of our model.

I try to optimize the model. Look further on train.csv, and I found that the tweets with positive sentiment are very much compared to the negative ones. So, I threw out some positive sentiment data to make the two sentiments equal in number (50:50).

Following is the classification report after that.

It is slightly better than before; maybe you know how to optimize the model, so the score is higher than mine.

Dump the pipeline model so I can use it in my application.

After building a Model for the classification task, I create the backend application to handle the server-side job of my web application. This backend purpose is to manage the database, crawling tweets, and process the classification task. I am new in Python, so correct me if there is a mistake (I work as a PHP developer).

This backend is a REST server that serves API and Service to our frontend, so there will be no fancy output in our application except data in a JSON form.

First, I make a virtual environment for my project. I read the Python virtual environment is a best practice when developing an application using Python. Virtual Environment will isolate the project environment so it won’t interfere with other projects.

packaging.python.org

Read the official website above to install, create, and activate the virtual environment.

My virtual environment named be-sentiment-app, you can name it whatever you want.

This is my backend folder structure. I have an application folder that contains three folders (helper, Models, Routes) and one Python file __init__.py. The others folder (bin, include, lib) are automatically created when creating the virtual environment.

Install Python packages that we need later using pip. I am not sure if this list completes or not. If there is a missing package, you can add it later.

This is the SQL script that I use in this backend. I use MySQL (MariaDB actually) as a database because it’s installed on my Computer. I named it database sentiment_app.

I copy the model which I dump before from Jupyter notebook (model_pipeline.pkl) into the helper folder because I want to use it later. After that, I created encoder.py and TweetClassifier.py inside the helper folder.

application/helper/encoder.py

encoder.py is a custom encoder to dumps data from which taken using SQL Alchemy into JSON form (I got this on StackOverflow).

application/helper/TweetClassifier.py

Tweet Classifier class is used for preprocessing and predict the sentiment of tweets. If you see, there are some code similarity that we use in Preparing.ipynb.

Back to the root of the application, I made these 4 Python files.

.env

I use dotenv files to store the credential and database connection configuration. Change the credential based on your Twitter API credential and change the database URI based on your current database URI.

config.py

This config file used to set flask and SQL alchemy configuration from our dotenv.

wsgi.py

start.sh,

these two files are used to run our Application later.

The entry script for this backend application is in __init__.py inside the application folder. This file determines what routes that must be used for every request.

application/__init__.py

I use one model for each table in the application/Models folder. These models are abstract for our table in the database used by SQL Alchemy to do database manipulation tasks.

application/Models/Hashtag.py

application/Models/Tweet.py

I also use one Route for each Model to make it easy to use. I created the Routes files in the application/Routes folder.

application/Routes/HashtagRoutes.py

application/Routes/TweetRoutes.py

These two routes files are used to do the logic on the backend; it handles the request and decides what to do after it.

The crawlTweet() function on TweetRoutes.py, which starts on line 69, is used to crawl the tweet data based on the hashtag parameter which passed by the frontend. For every tweet that crawled, this function does the classification task using the help of the helper/TwitterClassifier.py files (Line 160–162) and then saves it to the database

I only crawl 300 tweets in a single request of crawlTweet, you can change the number of maxTweets if you want.

This is the completed folder structure of be-sentiment-app

I got this folder structure from some tutorial and edited to Yii2 (PHP Frameworks) folder structure styles.

Our backend application doesn't have templates or view(on PHP terms) because we didn't need this. This application output is only a JSON format that will be consumed by the frontend application, not an end-user.

Run start.sh files to start the backend application.

To test this application, we can go to

http://127.0.0.1:5000/tweet/all?hashtag=iphonese&page=1

if you do this, of course, you will get the empty JSON because you don't have the Tweet contains iphonese hashtag. But if you have crawled the tweet using iphonese hashtag, it will show like this.

This is the URL list that we use to serve services for the frontend application.

Why I use Vue Js? Why I’m not using the Flask Standard Templating?

The answer is I want to learn more about Vue JS because there are high jobs demand with requirements of JS Frameworks like React or Vue.

To install Vue JS, you can read it on https://vuejs.org/v2/guide/installation.html.

I use vue-cli to create the Vue project

When installing, I prompted to check some preset, I choose manual then choose babel, linter, and router.

This is my folder structure after finish the installation process.

I install bootstrap-vue to beautify the user interface. BootstrapVue is a CSS framework based on Vue JS and Bootstrap 4, so I don't need to implement Bootstrap manually on my Vue Project.

To install, run this syntax on your Vue project folder.

You can read more about bootstrap-vue on this link (bootstrap-vue docs).

To implements the bootstrap-vue, I change the main.js inside src folder become like this:

src/main.js

I want to add the navbar on my application, so I change the App.vue files become like this:

src/App.vue

I don't have the navbar component yet, so I must create one in src/components/NavBar.vue

src/components/NavBar.vue

There are two link menus inside NavBar components, Home and Hashtag. Home is a page that contains the application information, and Hashtag is a page that contains the hashtag CRUD menu and Dashboard.

The following are the views that I used in fe-sentiment-app.

src/views/Dashboard.vue

The Dashboard.vue is a view that contains sentiment classification statistics of a hashtag.

src/views/Hashtag.vue

The Hashtag.vue is a view that contains the CRUD and Dashboard menu of Hashtags.

src/views/Home.vue

src/views/Tweets.vue

The Tweets.vue is a view that contains the Tweets of a hashtag.

Inside the view files above, there is some component that I used. The following are all of the components that I used.

src/components/ListHashtag.vue

Listhashtag.vue is a component that contains a table list of Hashtags.

src/components/SubMenuHashtag.vue

SubMenuHashtag.vue is a component that contains a submenu in the Hashtag view.

src/components/FormNewHashtag.vue

FormNewHashtag.vue is a component that contains a form to input and edit hashtag.

src/components/TopTweets.vue

TopTweets.vue is a component that contains a table list of 10 most retweeted tweets in a hashtag. This component is used in the Dashboard.

In the dashboard view, I use pie and line charts to make a beautiful graph. I use vue-chartjs to draw charts. To install vue-chartjs, run this command in the project folder

To know more about vue-chartjs, you can read the docs on this link (vue chart js guide and documentation)

src/components/LineChart.vue

src/components/PieChart.vue

This is my index.js inside src/router folder that defines what view that should be shown to the user.

src/router/index.js

This is the final result of my Application. To start the Vue Application, run this syntax

The home page of my application

Hashtag views

The default view of Hashtag is a List Hashtag Meny. You can see that there are five buttons in every hashtag.

The first one is the Fetch Data button, and this button is used to fetch the tweet based on a given hashtag.

When clicking this button, the application will crawl the last 300 tweets of a hashtag. You can edit this behavior on be-sentiment-app to change the maximum number, or you can add streaming function to get the tweet data in a realtime.

The New Hashtag and Edit menu are shared the same component.

View Tweets Button is used to view tweets of a hashtag.

You can see that many tweets are using a non-English language. Maybe you can optimize the model or change the crawling configuration to handle the non-English tweets.

Delete Button is to delete the hashtag.

Dashboard Button is to show the Dashboard View.

In the dashboard, I show the number of tweets, positive sentiments, and negative sentiments of a hashtag. There is a line chart that shows the number of sentiments day by day. You can see the proportion of the sentiments in a pie chart and see what is the Top Retweeted Tweet in a hashtag.

That’s all for this tutorial. Hopefully, it will be useful for all of us.

You can get all of my source code through the following GitHub link:

Thank You ❤️❤️👍👍